Scene Change Detection Based on Audio-Visual Analysis and Interaction

A scene change detection method is presented in this paper, which analyzes both auditory and visual information sources and accounts for their inter-relations and coincidence to semantically identify video scenes. Audio analysis focuses on the segmentation of the audio source into three types of semantic primitives, i.e. silence, speech and music. Further processing on speech segments aims at locating speaker change instants. Video analysis attempts to segment the video source into shots, without the segmentation being a ected by camera pans, zoom-ins/outs or signi cantly high object motion. Results from single source segmentation are in some cases suboptimal. Audio-visual interaction achieves to either enhance single source ndings or extract high level semantic information. The aim of this paper is to identify semantically meaningful video scenes by exploiting the temporal correlations of both sources based on the observation that semantic changes are characterized by signi cant changes in both information sources. Experimentation has been carried on a real TV serial sequence composed of many di erent scenes with plenty of commercials appearing in-between. The results are proven to be rather promising.